The present invention relates to a method and apparatus for decoding moving pictures coded by inter-frame coding with motion compensation, and more particularly to a method that reduces picture distortion caused by error propagation.
Moving pictures generate very large amounts of information. When transmitted over digital communication channels, they are usually compressed to reduce their data size by removing redundant information. Inter-frame coding is a method of removing redundancy in the temporal direction by coding the difference between the current frame and the preceding frame. Motion compensation is a refinement of inter-frame coding in which each part of the current frame is coded with reference to the nearby part it most closely resembles in the preceding frame, the location of this nearby part being indicated by a motion vector. Standard methods of inter-frame coding are given in recommendation H.263 of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T), and in the phase-two standard of the Moving Picture Experts Group (MPEG-2), which has been adopted by the International Standards Organization (ISO).
In the H.263 and MPEG-2 systems, a frame is divided into blocks of a certain size, referred to as macroblocks because they are made up of smaller blocks of luminance and chrominance information. A typical size is a square measuring sixteen picture elements or pixels horizontally and vertically. The macroblocks are further grouped into units variously termed groups of blocks (GOBs), slices, or video packets. The term xe2x80x98slicexe2x80x99 will be used herein to refer to any such grouping of macroblocks. FIG. 1 shows an example of a macroblock 2 and slice 4.
The coding process includes the use of variable-length codewords. Variable-length code synchronization words are inserted between the slices, so that if an error or data dropout prevents the boundaries of the variable-length codewords from being identified, correct decoding can resume in the next slice.
When a data error or dropout is detected, an attempt may be made to conceal the error by substituting data from a previous frame. In one known system, the macroblock or slice in which the error or dropout occurred is replaced with the corresponding macroblock or slice in the preceding frame. In FIG. 2, for example, erroneous or missing data 6 in frame n are replaced with data 8 from the same part of frame nxe2x88x921, which could be decoded correctly. The corresponding part of frame n+1 is then decoded with reference to the replacement data 8. In this case, since no motion occurs in the relevant parts 6, 8, 10 of frames nxe2x88x921, n, and n+1, the error is completely concealed.
A known decoding apparatus employing this method, shown in FIG. 3, comprises a bit-stream decoder 100, a prediction-error decoder 101, a motion-vector decoder 102, a picture predictor 103, an adder 104, a decoded-picture memory 105, and an error reporter 106.
The bit-stream decoder 100 in FIG. 3 decodes a received bit stream comprising variable-length codewords, and classifies the decoded values as transform coefficient information or motion-vector information. If it finds a variable-length codeword not listed in the corresponding coding table, or finds a synchronization codeword in an impossible location, it notifies the error reporter 106.
The prediction-error decoder 101 decodes the transform coefficient information to obtain prediction error values. For moving pictures coded according to the H.263 or MPEG-2 standard, for example, the prediction-error decoder 101 has the structure shown in FIG. 4, comprising a dequantizer 107 and an inverse transform processor 108. The dequantizer 107 obtains a so-called direct-current (DC) coefficient representing a mean value in a block or macroblock, and various coefficients representing higher spatial frequency components. If the DC coefficient is not within a certain range of values, the dequantizer 107 reports an error to the error reporter 106. The inverse transform processor 108 performs an inverse discrete cosine transform on the dequantized coefficients.
The motion-vector decoder 102 decodes the motion-vector information supplied by the bit-stream decoder 100 to obtain motion vectors. If the motion-vector decoder 102 detects an error, such as a motion vector pointing outside the frame, it notifies the error reporter 106.
The picture predictor 103 constructs a predicted frame by reading data from the preceding frame, which is stored in the decoded-picture memory 105, according to the motion vectors.
The adder 104 adds the prediction error values obtained by the prediction-error decoder 101 to the predicted frame obtained by the picture predictor 103 to obtain a decoded frame. The decoded frame is output as one frame of the decoded moving picture, and is stored in the decoded-picture memory 105, for use in decoding the next frame. The decoded-picture memory 105 has a capacity of only one frame.
If an error is detected by the bit-stream decoder 100, prediction-error decoder 101, or motion-vector decoder 102, the error reporter 106 directs the picture predictor 103 to read the slice in the preceding frame corresponding to the erroneous slice in the current frame from the decoded-picture memory 105, and insert this slice into the predicted frame supplied to the adder 104, replacing the erroneous slice. The error reporter 106 also directs the adder 104 not to add any prediction error values to the inserted slice. In the decoded frame, accordingly, the erroneous slice is replaced by the corresponding slice from the preceding frame.
A problem with this direct replacement scheme is that when motion is present, the inserted slice is visibly offset from the rest of the picture. If the motion vectors are large, the picture may be badly distorted. Furthermore, due to inter-frame coding, the distortion is passed on to subsequent frames, often becoming worse in the process. This is illustrated in FIG. 5, in which an error occurs in a part 12 of frame n with large motion vectors. When this part is replaced by the corresponding part 14 from the preceding frame (nxe2x88x921), the running figure appears to be torn apart, and the situation worsens in the next frame (n+1), as the replacement data reappear, without moving, in the corresponding part 16.
An object of the present invention is to reduce the distortion caused by error propagation in the transmission of moving pictures.
According to the present invention, when a moving picture coded by inter-frame coding with motion compensation is decoded, for each frame, at least two previously decoded frames of the moving picture are stored, together with information identifying any parts of these previously decoded frames that could not be decoded correctly. When the current frame is decoded, for each decodable part of the current frame, a motion vector indicating a correctly decoded part of one of the previous frames is calculated. A predicted frame is constructed by piecing together the parts indicated by the calculated motion vectors, and the current frame is decoded with reference to the predicted frame.
The calculation of the motion vector preferably starts from a received motion-vector value indicating a part of the most recent previously decoded frame. If this indicated part could not be decoded correctly, the motion vector is extended back into the next-most-recent previous frame, by multiplying the motion-vector value by a factor proportional to the additional elapsed time, for example, or by using stored values of previous motion vectors to retrace past motion.